Some Problems in the Evaluation of the Russian-German Machine Translation System MIROSLAV

In this article some of the problems encountered in designing an evaluation for the MIROSLAV machine translation (MT) system will be examined. We will concentrate mainly on the strategies employed in the evaluation and the evaluation areas, namely translation quality and translation productivity, whereas the former will be treated in more detail than the latter. In conclusion we will present two new methods for measuring translation quality through intelligibility and informativity. For the former we propose a novel sort of performance task, for the latter an automatic comparison of the semantic features assigned during the machine translation process to source and target texts. Both methods should reduce the subjectivity of judgments of translation quality. The former, because it measures intelligibility indirectly, that is, test persons are not required to make a direct judgment and the latter, because it will compare information units (semantic features) automatically.